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LARGE-SCALE  OPTIMIZATION  VIA  DISTRIBUTED  SYSTEMS 


Robert  R.  Meyer 

Center  for  the  Mathematical  Sciences  and 
Computer  Sciences  Department 
University  of  Wisconsin-Madison 

Summary 

Most  large-scale  network  optimization  problems  exhibit  structures  that  allow  the  pos¬ 
sibility  of  attack  via  algorithms  that  exhibit  a  high  degree  of  parallelism.  Such  structures 
include  quasi-independent  blocks  of  constraints  for  different  commodities  or  time  periods, 
and  geographically  disjoint  components  in  approximating  solutions.  The  emphases  of  our 
research  have  been  the  development  of  new  parallel  optimization  techniques  that  utilize 
these  and  related  fettures  in  order  to  take  advantage  of  distributed  computing  environ¬ 
ments.  We  have  also  undertaken  a  comparison  of  the  relative  efficiencies  of  approaches 
based  on  different  computer  architectures  such  as  messnge-pacsing  multicomputers  and 
shared-memory  multiprocessors.  The  parallel  algorithms  that  we  have  implemented  have 
made  possible  the  solution  of  extremely  large  linear  networks  (with  more  than  1  million 
variables)  and  nonlinear  network  optimization  problems  with  as  many  as  400.000  variables 
on  relatively  modest  parallel  computing  systems,  and  have  displayed  excellent  speedups 
relative  to  the  corresponding  single-processor  programs. 

1.  Research  Objectives  and  Accomplishments 

The  research  objectives  were 

1)  the  development  of  new  parallel  algorithms  for  the  solution  of  large-scale  nonlinear 
network  optimization  problems; 

2)  the  development  of  a  new  class  of  barrier  function  methods  applicable  to  multi- 
commodity  problems  in  order  to  utilize  the  nonlinear  techniques  of  objective  1)  in  con¬ 
junction  with  a  variant  of  interior  point  methods; 


3)  the  development  and  testing  of  parallel  algorithms  for  linear  generalized  network 
optimization  with  the  goal  of  achieving  high  efficiency  in  a  shared  memory  multiprocessor 
environment  for  problems  with  more  than  1  million  variables; 

4)  the  implementation  of  a  distributed  version  of  the  network  simplex  method  for 
linear  networks  with  hundreds  of  thousands  of  arcs: 

5)  the  testing  of  these  methods  on  representative  sets  of  large-scale  test  problems 

These  objectives  were  successfully  accomplished.  A  summary  of  the  results  obtained 

in  these  areas  is  given  below. 

The  research  carried  out  under  this  grant  on  parallel  algorithms  for  nonlinear  networks 
led  to  the  development  and  proof  of  convergence  of  a  class  of  new  methods  based  on  de¬ 
composition.  This  approach  involves  the  use  of  a  variety  of  block  separable  approximations 
to  decompose  problems  of  the  form: 

min  f(x) 
s.t.  =  bk 

h  <  <  >n  {k  =  l . K) 

(where  A 6*..,  I ^  and  are  the  constraint  data  for  commodity  k)  into  single-commodity 
linear  network  subproblems.  A  key  research  issue  resulting  from  this  form  of  decomposition 
is  the  distribution  of  work  between  the  '‘local  views'"  in  which  single-commodity  subprob¬ 
lems  are  solved  in  parallel  and  the  "global  view"  or  coordination  phase  in  which  data  from 
the  solution  of  many  such  subproblems  are  collected  and  modifications  of  these  solutions 
are  made  as  needed  in  order  to  achieve  improvements  in  the  original  or  barrier-augmented 
objective  function  (in  which  there  is  coupling  between  all  commodities).  In  joint  work  with 
my  Ph.D.  student  Gary  Schultz  (Schultz  and  Meyer  19S9],  we  have  generalized  the  earlier 
research  of  Chen  and  Meyer,  which  was  supported  by  this  grant,  to  allow  multi-dimensional 
searches  in  the  coordinating  task  and  to  incorporate  more  flexible  acceptance  criteria  in 
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the  updating  process.  Tests  of  this  approach  on  the  Patient  Distribution  System  problems 
of  the  Military  Airlift  Command  using  the  Sequent  Symmetry  S-Sl  multiprocessor  indicate 
at  least  am  order  of  magnitude  improvement  in  times  over  the  much  faster  KORBX  svstem. 

In  the  area  of  generalized  networks  (also  known  as  networks  with  gains  and  losses, 
since  flows  are  allowed  to  change  as  they  traverse  links),  we  have  developed  [Clark  and 
Meyer  1987]  and  [Clark  and  Meyer  13S9]  two  new  decomposition  methods  that  provide 
superlinear  speedups  for  certain  subclasses  and  extend  our  earlier  approach  to  large-grain 
problems  (i.e.,  problems  in  which  the  collection  of  quasi-trees  in  the  optimal  solution 
consists  of  a  small  number  of  components  each  of  which  has  a  large  number  of  nodes). 
These  approaches  have  been  implemented  on  the  Sequent  multiprocessor,  and  used  to 
solve  problems  with  more  than  1  million  variables. 

The  final  research  area  that  we  have  explored  is  the  parallel  implementation  of  the 
network  simplex  method  for  extremely  large  “pure"  (as  opposed  to  generalized)  linear 
networks.  The  challenge  in  this  area  was  to  utilize  appropriate  data  structures  so  that 
the  pricing  operation  could  be  parallelized  and  overlapped  with  the  pivoting  operation. 
We  have  obtained  superlinear  speedups  [Peters  19SS]  with  this  approach  on  the  Sequent 
multiprocessor,  testing  the  method  on  the  new  set  of  large-scale  network  test  problems 
recently  developed  by  Darwin  Klingman  and  John  Mote  (of  the  University  of  Texas  at 
Austin)  as  well  as  expanded  versions  of  thes-3  problems  with  up  to  500.000  arcs. 

2.  Publications 

R.  J.  Chen  and  R.  R.  Meyer:  “A  Scaled  Trust  Region  Method  for  a  Class  of  Con¬ 
vex  Optimization  Problems”.  Technical  Report  #675.  Computer  Sciences  Department. 
University  of  Wisconsin-Madison,  1987. 

R.  J.  Chen:  ‘‘Parallel  Algorithms  for  Convex  Optimization  Problems",  Ph.D.  thesis, 
Computer  Sciences  Department,  University  of  Wisconsin-Madison,  1987. 
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R.  Clark  and  R.  R.  Meyer:  ‘"Multiprocessor  Algorithms  for  Generalized  Network 
Flows",  Technical  Report  #739,  Computer  Sciences  Department,  University  of  Wisconsin- 
Madison,  19S7. 

J.  Peters:  “A  Parallel  Algorithm  for  Minimal  Cost  Network  Flow  Problems".  Technical 
Report  #762,  Computer  Sciences  Department.  University  of  Wisconsin-Madison.  19SS.  to 
appear  in  Networks. 

M.  D.  Chang,  M.  Engquist,  R.  Finkel  and  R.  R.  Meyer:  “A  Parallel  Algorithm  for 
Generalized  Networks",  Annals  of  Operations  Research  14.  125-145.  19SS. 

R.  J.  Chen  and  R.  R.  Meyer:  “Parallel  Optimization  for  Traffic  Assignment".  Mathe¬ 
matical  Programming,  Series  B  42,  327-345.  19SS. 

R.  R.  Meyer  and  S.  Zenios:  “Parallel  Optimization  on  Novel  Computer  Architectures". 
Annals  of  Operations  Research  14,  19SS. 

0.  L.  Mangasarian  and  R.  R.  Meyer:  “Parallel  Methods  in  Mathematical  Program¬ 
ming",  Mathematical  Programming  42,  19SS. 

G.  Schultz  and  R.  R.  Meyer:  “A  Flexible  Parallel  Algorithm  for  Block- Const  rained 
Optimization  Problems",  in  Im.pact  of  Recent  Computer  Advances  on  Operations  Research. 
R.  Sharda,  et  ah.  eds.,  Elsevier  Publishing,  19S9. 

R.  Clark  and  R.  R.  Meyer:  “Parallel  Arc-Allocation  Algorithms  for  Optimizing  Gen¬ 
eralized  Networks",  Technical  Report  #S62.  Computer  Sciences  Department.  University 
of  Wisconsin-Madison.  19S9.  (to  appear  in  Proceedings  of  Workshop  on  Supercomputers 
and  Large-Scale  Optimization). 

3.  Participating  Professionals 

Research  Assistants  (graduate  students  supported  under  this  grant): 

R.  J.  Chen  -  Ph.D.  in  Computer  Sciences  completed.  6/S7 
R.  Chirk  -  Ph.D.  in  Computer  Sciences  completed,  S/S9 
G.  Schultz  -  passed  Ph.D.  Screening  Examinations,  10/ SS. 


4 


4.  Interactions 


Invited  papers  presented  by  R.  R.  Meyer: 

10/S6  -  ORSA/TIMS  National  Meeting:  “Trust  Re  gion  Methods  for  Nonlinear  Net¬ 
works” 

4/S7  -  University  of  Maryland  Workshop  on  Scientific  Computing  Using  Parallel  Ar¬ 
chitectures:  "Parallel  Algorithms  for  Network  Optimization” 

5/S7  -  ORSA/TIMS  National  Meeting:  "A  Parallel  Trust  Region  Algorithm” 

5/S7  -  SIAM  Conference  on  Optimization:  "Parallel  Algorithms  for  Large-Scale  Non¬ 
linear  Networks” 

6/S7  -  NATO  Advanced  Research  Workshop  on  Algorithms  and  Problem  Formulations 
in  Mathematical  Programming:  “Modelling  for  Parallel  Optimization" 

S/S7  -  Symposium  on  Parallel  Optimization:  “Parallel  Algorithms  for  Large-Scale 
Network  Optimization” 

12/S7  -  Third  SIAM  Conference  on  Parallel  Processing:  “Parallel  Optimization  of 
Large-Scale  Linear  Networks” 

4/SS  -  ORSA/TIMS  National  Meeting:  "Parallelizing  the  Network  Simplex  Method' 

5/SS  -  University  of  Minnesota  Workshop  on  Supercomputers  and  Large-Scale  Opti¬ 
mization:  “Parallel  Computing  for  Large-Scale  Network  Optimization" 

S/SS  -  13th  International  Symposium  on  Mathematical  Programming  -  Tokyo:  "Par¬ 
allel  Algorithms  for  Large-Scale  Network  Optimization  Problems".  "Parallel  Optimization 
of  Nonlinear  Networks" 

10/S8  -  ORSA/TIMS  National  Meeting  -  Denver:  “Parallel  Algorithms  for  Traffic 
Assignment” 

1/S9  -  ORSA  (Comp.  Sei.  Tech.  Section)  Conference  on  Impact  of  Recent  Computer 
Advances  on  Operations  Research  -  Williamsburg:  "Parallel  Computation  in  Large-Scale 
Network  Optimization” 
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1/S9  -  Programacion  Matematica  'S9  -  Madrid:  "Parallel  Algorithms  for  Linear  Net¬ 
work  Flows",  “Parallel  Algorithms  for  Nonlinear  Network  Flow  Optimization" 
Interactions  with  AFOSR: 

3/S7  -  Visited  by  Dr.  Samuel  Rankin,  AFOSR,  to  discuss  research  directions  relevant 
to  an  AFOSR  initiative  in  parallel  optimization 

4/S7  -  Visit  to  Mathematical  and  Information  Sciences  Directorate.  AFOSR.  to  discuss 
research  directions  relevant  to  parallel  optimization 
Other  Consulting  and  Advising: 

12/S6  -  Member  of  NSF  site  visit  team  evaluating  a  University  of  Michigan  proposal 
to  the  Coordinated  E.xoerimental  Research  Program 

9/S7  -  Member  of  NSF  site  visit  team  evaluating  University  of  Minnesota  and  Uni¬ 
versity  of  Colorado  Coordinated  Experimental  Research  Programs 

Reviewer  for  numerous  AFOSR  and  NSF  grant  proposals. 

5.  Other  Professional  Activities  Related  to  Research 

Co-editor  of  proceedings  of  AFOSR-sponsored  Symposium  on  Parallel  Optimization. 
Mathematical  Programming,  Series  B,  vol.  42,  no.  2.  19SS,  (see  Appendix). 

Co-editor  of  Committee  on  Algorithms  Newsletter  (Math.  Programming  Society). 

Area  Editor.  Parallel  Computation.  Journal  on  Computing  (a  new  journal  of  tire 


Operations  Research  Society). 
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SYMPOSIUM  ON  PARALLEL  OPTIMIZATION 

Olvi  L.  Mangasarian 

Robert  R.  Meyer 

Computer  Sciences  Department 
University  of  Wisconsin 

1210  West  Dayton  Street 
Madison,  WI  53706 

With  AFOSR  and  ONR  support  the  following  eighteen  papers 
were  presented  at  the  Symposium  on  Parallel  Optimization  held  in 
Madison,  August  10-12,  1987: 

George  B.  Dantzig:  "Planning  under  uncertainty  using  parallel 
computing" 

Stavros  A.  Zenios  &  John  M.  Mulvev:  "Nonlinear  network  program¬ 
ming  on  vector  supercomputers" 

Dimitri  P.  Bertsekas:  "Parallel  relaxation  methods  for  linear 
network  flow  problems" 

Ron  S.  Dembo  &  Partrick  Hdnaff:  "Vectorized  conjugate  gradient 
projection  methods: 

Shih-Ping  Han:  "Parallel  quasi-Newton  algorithms  for  nonlinear 
programming" 

Michael  D.  Chang,  Michael  Engquist,  Raphael  A.  Finkel  &  Robert 
R.  Meyer:  "Solving  multiperiod  generalized  network  problems 
on  a  parallel  computer" 

Robert  R.  Meyer:  "Parallel  algorithms  for  large-scale  network 
optimization" 

Alexander  H.G.  Rinncov  Kan:  "Parallel  partitioning  methods" 

J.  Ben  Rosen  &  Andrew  Phillips:  "Multitasking  and  parallel 
algorithms  for  linearly  constrained  global  optimization" 

Klaus  Ritter:  "On  the  use  of  parallel  computing  and  automatic 
differentiation  in  mathematical  programming" 

Robert  Schnabel,  Richard  H.  Byrd  &  G.  Shultz:  "Speculative 

function  evaluation  in  parallel  optimization"  j 

i.ctir  .  t  ai.aiiei  ai.a  bluCK-iw3i.duiv6  uu^Li*odo  with  appli¬ 

cations  in  medical  imaging  and  radiation  therapy" 

Jong-Shi  Pang:  "Parallel  Newton  methods  for  complementarity 
problems " 

Lucio  Grandinetti:  "Applications  of  vector  computers  to 
optimization  of  1 argp  scale  systems" 

A.  Ech-cherif,  Joseph  G.  Ecker  &  M.  Kupf ersunruid :  A  ^ai_ailei 

algorithm  for  nonlinear  programming  on  the  hypercube" 

Jares  K.  Ho:  "Linear  programming  decomposition" 

Danny  Sorensen:  "Programming  methodology  and  performance  issues 
for  advanced  computer  architecture" 

Olvi  L.  Mangasarian  &  Renato  De  Leone:  "Serial  and  parallel 

successive  overrelaxation  in  linear  and  quadratic  programming" 

This  symposium,  the  first  of  its  kind  in  the  field  of 
optimization,  addresses  the  very  important  and  promising  area 
of  parallel  computation.  Sixty-three  persons  attended  tne 
symposium.  Proceedings  of  the  symposium  will  be  published  as 
a  special  issue  of  the  journal  Mathematical  Programming  Studies. 
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